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four categories indicating the extent to which the knowledge/academic 
skills needed to answer each item correctly is relevant to competent 
performance as a beginning teacher of ECE in New York City; and (2) a 
knowledge estimation panel that reviewed each iter, and responded in 
one of eight categories reflecting the individual item difficulty for 
persons who have the minimum levels of knowledge/academic skills 
needed for competent performance as a beginning teacher of ECE in New 
York City. Job relevance findings were reviewed by two statistical 
methods; 96.0% or 99.3%, respectively, of the items were found to be 
job relevant. Two analyses of results from the knowledge estimation 
panel resulted in scores of 500 and 510, respectively, as potential 
passing marks; 500 was the recommended passing score, partly because 
the statistical methodology agreed with that used in setting the 
passing score on the NTE Core Battery. The content validity of the 
test was deemed acceptable. The methodology in this study could serve 
as a prototype for studies of other specialty area tests. Fourteen 
data tables and 15 appendices are included. (Author/SLD) 
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EXECUTIVE SUfflARY 



The New York City Board of Education uses the National Teacher Examina- 
tions' (/TE) Core Battery and Specialty Area Tests as part of the alternative 
process for teacher licensure under the New York State Decentralization Law 
2590-J. Prerequisite to the use cf NTE Tests, the test publisher - toe Educa- 
tional Testing Service (ETS) of Princeton, New Jersey - requires clients to 
validate each of the tests. Until now, the NTE Specialty Area Test in Early 
Childhood Education has not been validated for use in New York City public 
schools. 

This study provides evidence of the validity of the NTE Early Childhood 
Education test as it is used by the New York City school system. In addition, 
based upon expert judgments provided by panels of New York City educators* a 
potential passmark is recommended. This passmark is an estimate of the minimum 
level of knowledge and skills required for competent performance by beginning 
Teachers of Early Childhood Education in New York City public schools. 

The NTE Early Childhood Education test is designed to measure the knowledge 
and academic skills required by entry-level teachers of Early Childhood Educa- 
tion. This is a 150-item multiple choice test with a time limit of two hours. 
Various forms of this test are available. Consistent with E T S policy, only one 
form of the NTE Specialty Area Test in Eirly Childhood Education was reviewed 
in this study. 

The research methods used in this study are based upon those implemented 
previously by ETS in studies commissioned by the New York State Education 
Department. Furthermore, these methods are consistent with federal ''Uniform 
Guidelines 11 and psychometric standards of various professional organizations. 

In this study, judgments were provided by 32 New York City public school 
educators with expertise in Early Childhood Education. Each individual par- 
ticipated in two separate professional panels. First, the Job Relevance Panel 
reviewed each item and responded in one of four categories indicating the extent 
to which the knowledge or academic skills needed to answer each item correctly 
is relevant to competent performance as a beginning Teacher of Early Childhood 
Education in New York City. Subsequently, the Knowledge Estimation Panel 
reviewed each item .and responded in one of eight categories reflecting the 
difficulty of individual items for persons who have the minimum levels of 
knowledge and academic skills necessary for competent performance as beginning 
Teacher of Early Childhood Education in New York City. 

Consistent with previous studies, Job Relevance Panel member responses 
were statistically analyzed by two methods. Depending upon the statistical 
method, either 96.0% or 99.3% of the items were found job relevant . Subsequent- 
ly, results from the Knowledge Estimation Panel were statistically analyzed and 
converted to the reporting scale established by ETS. Rounded mean scaled scores 
of 500 and 510, respectively, resulted from the two statistical methods. 
Reliability estimates were obtained and reported. 
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The Method I mean score of 500, rather than the score of 510 derived from 
Method II, is recommended as a potential passmark. This recommendation is based, 
in part, upon the fact that the NTE Core Battery passmarks established by the 
New York State Education Department were obtained using Method I analyres. The 
potential passmark may be modified by taking into account the Standard Error of 
Measurement of this particular test. Modification of the passmark may be 
required due to "supply and demand" and other factors specified in this report. 

In conclusion, based upon methods and criteria established in previous 
validity studies, the results of this study demonstrate the content validity of 
the NTE Specialty Area Test in Early Childhood Education for use as one part of 
the license requirements for beginning Teachers of Early Childhood Education in 
New York City public schools. Furthermore, an appropriate passmark was recom- 
mended. It is noted, finally, that the methodology and statistical analyses of 
this study could ser*e as a prototype for studies to assess the validity of, and 
establish New York City passmarks for, other NTE Specialty Area Tests. 
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I. INTRODUCTION 



The intent of this study is to assess the validity of the 
NTE Specialty Area Test in Early Childhood Education for use by 
the New York City school system. The two components of this 
study are: 1) content review of the test as it is related to the 
entry-level job requirements of New York City Teacher ot Early 
Childhood Education, and 2) identification of potential criteria, 
standards or passmarks which reflect minimally acceptable 
performance by entry-level New York City Teachers of Early 
Childhood Education. 

This study is necessary because the NTE Specialty Area Test 
in Early Childhood Education is required by the New York City 
school system. The Guidelines for Proper Use of NTE Tests 
(Educational Testing Service, 1985a) specify that validity and 
standard setting studies must be conducted as prerequisites for 
the use of NTE tests. However, the Early Childhood Education 
test has not been validated for usa in New York. Consequently, 
the Educational Testing Service has apprised the New York City 
Board of Education that such a study must be conducted. 

USE OF THE NTE SPECIALTY AREA TESTS IN NEW YORK CITY 

A passing score obtained on the NTE Specialty Area Test in 
Early Childhood Education is an essential criterion in one of the 
routes by which an individual may become licensed to teach pre- 
kindergarten through grade two in some New York City public 



elementary schools (New York City 3oard of Education, 1986; 
1989a). This is an alternative rou^e for teacher licensure, in 
compliance with the Now York City School Decentralization Law 
2590- j (McKinney's Consolidated Taws of New York - Annotated, 
1981) • Currently, this NTE Specialty Area Test route for 
licensure applies only to those New York City public elementary 
schools which are ranked in the bottom 45 percent on the annually 
administered citywide reading test (see, for ranking of schools, 
New York City Board of Education, 1989b) . 

In general, the primary route for teacher licensure in New 
York City includes satisfactory performance on a test 
administered by the Board of Examiners (see, for examination 
announcement; Board of Examiners, 1989). It is noted, however, 
that currently proposed legislation may provide the basis for NTE 
Specialty Area Tests to replace the written component of some 
Board of Examiners tests. Should this legislation be 
V "lemented, the NTE Specialty Area Tests might be used as a 
written component of procedures which establish pools of teacher 
candidates in several disciplines, including Early Childhood 
Education. 

The New York State Education Department has demonstrated the 
validity of several of the NTE Specialty Area Tests (Educational 
Testing Service, 1985b; 1985c; 1987a). However, these studies 
have not included the test for Early Childhood Education. As of 
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this time, furthermore, none of tlv* NTE Specialty Area Tests have 
been adopted for use in New York State. Consequently, New York 
State passmarks have not been mandated for any of these tests. 
It is noted, however, that the New York State Education 
Department has: demonstrated the validity of the NTE Core Battery 
(Educational Testing Service, 1983); adopted the three NTE Core 
Battery tests; and established NTE Core Battery passmarks for use 
in New York Sta^e. 

The manner in which NTE passmarks are established is defined 
by New York State law. Specifically, under Chapter 628 of the 
Laws of 1989, signed by Governor Cuomo on July 21, 1989, the New 
York State Commissioner of Education has the responsibility of 
setting NTF passmark scores based upon appropriate validity 
studies. Where the Commissioner has not established such 
passmarks, the Chancellor of New York City public schools may set 
the passmarks based upon validity rtudies. 

THE NTE SPECIALTY AREA TEST IN EARLY CHILDHOOD EDUCATION 

The NTE Specialty Area Tests are two-hour tests designed to 
measur-. the knowledge and academic skills generally acquired by 
examinees in preparing for teaching careers in specific fields. 
These tests focus on areas covered by most teacher education and 
training programs. The examination in Early Childhood Education, 
designed for examinees who have completed their undergraduate 
preparation in this field, is intended to measure academic 

3 
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competence in Early Childhood Education. As delineated in Educa 
tional Testing Service (1987b) , this test focuses on two aspects 
1) knowledge and understanding of the nature of growth, 
development, and learning of young children; and 2) ability to 
judge the appropriateness of teaching behaviors, curriculum 
organization, and activities for children of ages three through 
eight, as related to their growth, development and learning. A 
more detailed description of this 150 item test is presented in 
Table 1. 



TABLE 1 



Test Content Description 
Early Childhood Education 



The examination in Early Childhood Education is intended primarily 
for candidates who have completed their undergraduate preparation in 
the field. In keeping with trends in the field, the examination 
focuses on the candidates' understanding and recognition of 
appropriate applications of early childhood education knowledge and 
theory related to the development of the whole child. Major 
dimensions of the examination include the following areas: 

APPROXIMATE 

CONTENT TOPICS % OF TEST 

I. The nature of growth, development, and learning of 30% 

young children - including aspects of cognitive 
development (concepts, skills, language), physical 
development, and personal -social development 

II. Factors that influence growth and development - 10% 

including biological , familial, nutritional- 
hygienic, and cultural factors 

III. Tne contributions of developmental and curriculum 12% 

theory to early childhood education practices - 
including major streams of developmental theory 
(e.g., cognitive, behaviorist, social-learning) 
and major streams of curriculum theory (e.g., Bank 
Street, Bereiter-Engiemann, Kamii, Montessori) 

IV. The planning and implementation of curriculum - 29% 

including the planning, selection, and implemen- 
tation of appropriate curriculum experiences; 
management of the physical learning environment; 
behavior management issues and practices; and 
utilization of family-community resources in the 
learning program 

V . The evaluation and reporting of student progress - 1 3% 

including the selection and use of ~ormal and 
informal assessment instruments or procedures, the 
maintenance of records of a child's progress, and 
effective communication with parents about a child's 
total developmental progress 

V I . Profe ss ional and legal responsibilities of the early 6% 

childhood teacher - including effective teacher 
interaction with other adults in the learning setting 
and cognizance of legal regulations that impact on 
teaching in the early childhood setting 



Used with copyright permission of the Educational Testing Service. 



II. OBJECTIVES AND METHODOLOGY OF THIS STUDY 



The objectives of this study are: 

1) To assess the validity of the content of the NTE 
Specialty Area Test in Early Childhood Education to 
relevant job requirements of entry-level Teacher of 
Early Childhood Education in New York City public 
schools , and 

2) To recommend a passing score (or provide options from 
which policy -makers select one score) on the NTE 
Specialty Area Test for Early Childhood Education. 

The passing score will reflect the minimum level of 
knowledge required by entry-level Teachers of Early Childhood 
Education in New York City. It is anticipated that policy- 
makers will use the recommendation to establish the minimum 
qualifying score or passmark required as one of the criteria for 
licensure* In addition, other policy related issues, such as 
"supply and demand" and affirmative action goals, may be taken 
into consideration* 

TASKS AND METHODOLOGY 

An overview of the tasks which comprise this project is 
presented in Table 2. No single process leading to the setting 
of standards for certification is generally acknowledged by 
professional educators and meisurement specialists to be either 
the best or the only defensible method for conducting this type 
of study. The methods used in this particular study are based 
upon those implemented previously by the Educational Testing 
Service (1983; 1985b; 1985c; 1987a) in validity studies 
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TABLE 2 
Overview of Project Tasks 

Community School District Superintendents nominate teacher and 
supervisor candidates for professional panels 

Select and confirm participation of panel members 

Prepare test materials for professional panel use 

Develop data collection materials including: instructions to 
panel members , job relevance form, knowledge estimation 
form, and related comment sheets 

Proof-read and print panel instructions, response forms and 
comment sheets 

Prepare background information sheets, rosters, payroll forms and 
site packets for panel members 

Conduct Panel meetings 

Perform quality control on forms 

Scan forms 

Analyze data from Job Relevance and Knowledge Estimation Panels 
Prepare tables with panel data 

Obtain nationwide performance data from Educational Testing 
Service for NTE Early Childhood Examinations 

Relate panel judgments to actual test data 

-Draft report 

Review report 

Submit final report and executive summary 
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commissioned by the New York State Education Department. In 
addition, these methods are in accordance with accepted 
measurement guidelines (e.g., American Educational Research 
Association, et al. . 1985) and federal "Uniform Guidelines" 
(Equal Employment Opportunity Commission, et al. . 1978). A 
survey of selected literature dealing with content validity 
studies, standard setting, and validity-study related issues can 
be found in Educational Testing Service (1985c, Chapter II). 

DIFFERENT FORMS OF THE TEST 

Currently, there are three forms of the NTE Specialty Area 
Test in Early Childhood Education and a fourth form xs being 
developed. In past validity studies, panels reviewed items from 
several forms of a given subject area test. However, the 
Educational Testing Service currently supplies only one form of a 
test for validity studies. Consequently, only one form of the 
Early Childhood Education test was reviewed. 
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III. THE JCB RELEVANCE AND KNOWLEDGE ESTIMATION PANELS 



The suitability of using the NTE Specialty Area Test in 
Early Childhood Education was assessed by a group of 32 
experienced New York City educators who specialize in Early 
Childhood Education. Each of these educators provided their 
expert judgments as members of both of the following two panels: 
(1) the Job Relevance Panel which judged the relevance of the 
content of the test questions to the job of beginning Teacher of 
Early Childhood Education in New York City; and (2) the Knowledge 
Estimation Panel which estimated the test performance required of 
beginning Teacher of Early Childhood Education in New York City, 

ESTABLISHING THE PANELS 

Several factors were taken into consideration when the size 
and nature of the membership of the two panels was finalized. 
These factors included: 

(a) designation of individuals who are currently serving in 
New York City public schools as practicing 
professionals including teachers, supervisors and 
administrators of Early Childhood Education; 

(b) representation from each New York City Community 
School District; 

(c) representation from the Early Childhood Education Unit 
at headquarters of New York City public schools; 

(d) reasonable representation of diverse racial and ethnic 
groups ; and 

(e) the need to obtain sufficiently reliable judgments 
about each question in the test. 
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On the basis of these considerations, it was decided to 
invite representatives from each of New York City'*: 32 
decentralized Community School Districts. In addition, educators 
from the New York City public school headquarters' Early 
Childhood Education Unit were invited to participate. 

In May 1989, a memorandum (see Appendix A) was sent from 
Dr. Dolores M. Fernandez, former Deputy Chancellor for 
Instruction and Development, to all New York City Community 
School District Superintendents. This memorandum invited each 
District Superintendent to assign one Early Childhood Education 
representative to participate as a panel member in this validity 
study. 

Of the 32 New York City Community School Districts, 28 or 
87.5 percent had representatives in this study. Only school 
districts 4, 8, 21 and 31 did not participate. It is noted, 
also, that four professional educators from the headquarters 
Early Childhood Education Unit also participated as panel members 
in this study. 

All panel members were required to complete a "Background 
Information Sheet 11 (see Appendix B). In addition, panel members 
were requested to provide authorization for their na*~s to be 
listed as panel members in the final study report (see appendix 
C). It is noted that all panel members provided such 
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authorization. Consequently, each panel member's name and 
position are listed in Table 3. 

DESCRIPTION OF PANEL MEMBERS 

The same professional educators provided expert judgments 
on both panels. Of these, 28 or 87.5% were employed on a full- 
time basis in one of New York City's 32 decentralized Community 
School Districts. The additional four or 12.5% of the panel 
members were employed by the Early Childhood Education Unit at 
headquarters of the New York City public school system. 

All 32 panel members were female. Panel member age and 
ethnicity summary data are presented in Table 4. Of the 75 
percent of panel members who provided ethnicity data, the largest 
group (i.e., 46.9%) were white and the second largest group 
(i.e., 25%) of panel members were black. Furthermore, the 
largest group (i.e., 50%) of panel members were in the 41 to 48 
year age category. 

Panel members were asked their "Total years of teaching 
experience . " Their responses to this question had a mean of 
approximately 21 years (with a standard deviation of 7.32) and a 
median of 20 years (with a range from four to 40 years) . The 32 
panel members reported their highest earned degrees as: 
doctorates (3 or 9.4%), masters (28 or 87.5%) or bachelors (1 or 
3.1%) degrees. 
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TABLE 3 



Early Childhood Education Professional Panel 



ASSIGNMffl 




PANEL MEMBER NAME 


POSITION 


District 


1 


/~t _ „ J * _____ 1 

Geri Appel 


Early Childhood Coordinator 


District 


I 


Sandra Yark 


Early Childhood Coordinator 


uiscricc 


o 


htta rrosnansKy 


Director, Early Childhood Education 


District 




Juanita Johnson 


Assistant Principal Intern 


District 


c 

0 


Rosemarie Crownover 


Early Childhood Coordinator 


District 


/ 


Bea Brown 


Early Childhood Coordinator 


District 


Q 


Jacquelyn Tatum 


Staff Developer, Project Child 


District 


10 


Rachel Floras 


Staff Developer, Early Childhood 


uisuricu 


11 


Kania Gold 


Principal, Elementary School 


District 


1 0 
LL 


Joan Salvatore 


Early Childhood Coordinator 


uiscricc 


1 J 


Carole Friedman 


Program Director, NYS Pre-Kindergarten 


District 


14 


Mary L. Driscoll 


Director, Early Childhood Program 


District 


ID 


Eileen Jones 


Early Childhood Coordinator 


District 


lb 


Renee Gregory 


Assistant Dist. Admin. Special Education 


District 


1 / 


Delores Theobald 


« „1 ^ 1 Jl _ J *m mm . 

Early Childhood Coordinator 


uistrict 


1 ft 
10 


ueura uratnwaite 


Early Childhood Coordinator 


L/is trie t 


1 Q 


bnaron oimraons 


Supervisor, Early Childhood Education 


p\ 4 e f- v* 4 r» f- 

L/istricc 




Rosemarie Papa 


Early Childhood Coordinator 


UlS Li -L L* L» 


99 


fvaurii.een Lavin 


Assistant Principal, Elementary School 


District 


23 


Alice Gottlieb 


Early Childhood Coordinator 


District 


24 


Christine Sobocienska 


Early Childhood Coordinator 


District 


25 


Marilyn wunaeT 


Ear?y Childhood Coordinator 


District 


26 


Caren Hirsch 


Early Childhood Coordinator 


District 


27 


haytne Nelson 


Starr Developer, Project Child 


District 


28 


Betty Silver 


Chairperson, Early Childhood Committee 


District 


29 


Adele Armstrong 


Teacher Trainer, Early Childhood 


District 


30 


Anne Geller 


Consultant, NYS Kindergarten Program 


District 


32 


Margaret Connolly 


Supervisor, Early Childhood Education 


Headquarters 


Marjorie McAllister 


Director, Early Childhood Education 


Headquarters 


Eleanor Ukoli 


Assistant Director, Early Childhood 


Headquarters 


Toby Rusgo 


Supervisor, Early Childhood Education 


Headquarters 


Carmela Cohen 


Supervisor, Eatly Childhood Education 
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TABLE 4 



Professional Panel Members' Age and Ethnicity 





Aae GrouD 




Ethnicitv 


Cateaorv 


Numbar 


Percent 


Cateaorv 


Number 


18-25 


0 


0 


American Indian 


0 


26-32 


0 


0 


Asian 


0 


33-40 


6 


18.8 


Black 


8 


41-48 


16 


50.0 


Hispanic 


1 


49-56 


4 


12.5 


White 


15 


57+ 


4 


12.5 


No Response 


8 


No Response 2 


6.3 







Percent 

0 
0 

25.0 
3.1 
46.9 
25.0 
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The "Degree Major" reported by the J2 panel members is 
summarized as: Early Childhood Education (14 or 43.8%), 
Supervision and Administration (5 or 15.6%), Elementary Education 
(3 or 9.4%), Education, English or Reading (4 or 12.5%) and "No 
Response" (6 or 18.8%). 

As presented earlier, Table 3 lists each panel member's 
name, position and assignment. The positions of panel members 
are summarized by category in Table 5. Review of Table 5 
indicates that most panel members (53.1%) are Early Childhood 
Supervisors and/or Coordinators. In addition, the Board of 
Examiners 1 related licenses held by panel members are summarized 
in Table 5. Most panel members (a total of 56.2%) hold licenses 
as Supervisors or Teachers of Early Childhood Education. 

ESTABLISHING THE HALF PANELS 

To examine tne consistency of panel judgments, responses 
from the Job Relevance and Knowledge Estimation Panels were each 
split into half panels. The results from each of the half panels 
were statistically analyzed. Assignment of panel members into 
the separate half panels was based on the following criteria: 

1. Each New York City Community School District 
represented at the panel meeting was rank- 
ordered by district performance on the 1988 
annual Citywide Reading Test (i.e., the 
Degrees of Reading Power Test) . 
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TABLE 5 



Professional Panel Members 1 Positions 
and Related Licenses 



Present Position 


Number 


Percent 


Early Childhood Supervisor/Coordinator 


17 


53 . 1 


Director 


3 


9.4 


Staff Developer 


3 


9.4 


Assistant Principal (A. P.) or A. P. Intern 2 


6.3 


NYS Kindergarten Program Staff 


2 


6.3 


Assistant Director 


1 


3.1 


Principal 


1 


3.1 


Teacher Trainer 


1 


3.1 


Assistant District Administrator 


1 


3.1 


Chairperson, Early Childhood Committee 


1 


3.1 


Related License 


Number 


Percent 


Supervisor Early Childhood Education 


9 


28. 1 


Teacher Early Childhood Education 


9 


28. 1 


Teacher Common Branches 


6 


18.8 


Teacher Elementary Education 


2 


6.3 


Director Early Childhood Education 


1 


3.1 


Teacher Special Education 


1 


3.1 


Educational Administration 


1 


3.1 


Teacher English 


1 


3.1 


No Response 


2 


6.3 
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2. Alternate members of the rank-ordered list 
weie assigned to each half panel. 

3. The four Board of Education headquarters 
personnel from the Early Childhood Education 
Unit were randomly assigned, two each, to 
each of the half panels. 

4. The total years of teaching experiences of 
each half panel were summed and compared. To 
make the sums approximately equal, two 
members were interchanged between half 
panels. 

For each of the half panels, the number of members and their 
respective years of teaching experience are depicted in Table 6. 
For both the Job Relevance and Knowledge Estimation Panels, Half 
Panel 1 contained the same members. Half anel 2, however, 
differed by one member. This particular member arrived late and, 
consequently, attended only part of the Job Relevance session and 
the entire Knowledge Estimation session. As such, judgments from 
this particular member were obtained only for the Knowledge 
Estimation Panel. 

The reason for rank-ordering by school district standardized 
reading test performance was because of the possibility that 
panel members 1 expectations of a minimally knowledgeable 
beginning teacher may be affected by the standardized achievement 
test performance of the pupils in their respective school 
districts. In addition, panel members were rar> "-ordered by years 
of teaching experience because of the possibility that such 
experience might influence panel members 1 expectations. By 
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establishing half panels based upon both school district test 
performance and panel members 1 years of professional experience, 
more comparable or "better matched" half panels were anticipated. 
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TABLE 6 

Characteristics of Half Panels 



Job Relevance Half Panels 

Half Panel 1 Half Panel 2 

Number of Members 16 15 

i unbined Teaching 

Experience (years) 324 325 

Mean Teaching Experience (years) 20.25 21.67 



Knowledge Estimation Half Panels 

Half Panel 1 Half Panel 2 

Number of Members 16 16 

Combined Teaching 

Experience (years) 324 341 

Mean Teaching Experience (years) 20.25 21.31 
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IV. MEETING OF THE JOB RELEVANCE AND KNOWLEDGE ESTIMATION PANELS 



On June 16, 1989, the Professional Panel met at 110 
Livingston Street. The agenda for this meeting can be found in 
Appendix D. 

WELCOME AND ORIENTATION 

Panel members were welcomed by the Director of the Office of 
Research, Evaluation and Assessment, Mr. Robert Tobias. 
Subsequently, panel members were addressed by two professionals 
from the Board of Education's Division of Human Resources: Mr. 
Roberto Fuentes (Administrator) and Ms. Zenaida Tejada (Director 
of the Teacher Certification Unit). Mr. Fuentes and Ms. Tejada 
indicated the importance of the NTE route for staffing New York 
City public schools. 

Orientation for the overall study was provided by the 
Project Director, Dr. Gary Kippel. He specified the objectives, 
nature and importance of this particular validity study. He 
provided, also, a brief explanation of the concept of validity. 
Folders containing various materials required for this study were 
distributed to each panel member. Panel members were told that 
in all instances they would make their judgments independently 
and that there would be no attempt to obtain a consensus of 
j udgment • 
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TEST SSWFITY 

Test security was a primary concern. At all times, 
appropriate procedures were implemented to ensure optimal test 
security. Both the Job Relevance and Knowledge Estimation Panels 
reviewed a secure form (with correct answers indicated) of the 
NTE Specialty Area Test in Early Childhood Education, At the 
beginning of the meeting, all panel members were required to read 
and sign an "NTE Disclosure Policy and Release Form" (see 
Appendix E) . 

To insure test security, furthermore, the Educational 
Testing Service assigned their representative, Dr. David Goldman, 
to deliver and retrieve the tests and, also, to monitor all 
phases of the panel meetings. This ETS representative asked that 
no one take notes, record, copy or disclose items during the 
study. He requested, also, that anyone leaving the room raise 
their hands so that he could collect the test books before they 
leave the room. In effect, no opportunity was provided for 
anyone to copy test questions. Each test book was coded with a 
unique identification number for test security purposes. At the 
end of the study, the monitor counted all the tests and verified 
that each test was returned to him. 
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V. TEST CONTENT AND JOB RELEVANCE 



The Job Relevance Panel was established to judge the extent 
to which knowledge or skills tested by individual questions is 
appropriate to the job requirements of an entry-level Teacher of 
Early Childhood Education in New York City. The Job Relevance 
Panel consisted of 31 members. One additional person arrived too 
late to participate in this panel. Subsequently, this individual 
did participate in the Knowledge .Estimation Panel which was 
comprised of a total of 32 members. 

JOB RELEVANCE PANEL PROCEDURES 

First, participants were familiarized with this test. 
Specifically, each panel member was provided with a "Test Content 
Description" (see Appendix F) which contained the specifications 
describing the NTE Specialty Area Test in Early Childhood 
Education. Included in the test specifications were the 
percentage and number of items in each of the topics covered by 
this test. 

Second, tasks to be performed by the Job Relevance Panel 
were discussed. Subsequently, panel members were asked to read 
the "Overview of Tasks..." (see Appendix G) and "Instructions for 
completing the Job Relevance Form" (see Appendix H). 



ERIC 



21 

32 



Next, panel members were provided the: NTE Specialty Area 
Test in Early Childhood Education , machine-scannable "Job 
Relevance Form " (see Appendix I) , sharpened number two lead 
pencils, "Panelist's Comment Sheet" (see Appendix J) , and "Test 
Comprehensiveness Comment Sheet" (see Appendix K) . Panel members 
were provided an opportunity to ask questions. Finally, they 
were asked to provide their judgments of the test questions and, 
also, to complete the comment sheets. The comment sheets were 
intended to obtain feedback on individual test items. 

Job Relevance Panel members were asked to judge whether each 
test item or question was relevant to competent performance 
required of the job of a beginning Teacher of Early Childhood 
Education. For item judgments, panel members were asked to 
select one of the following four judgment categories on the 
machine-scannable "Job Relevance Form" (see Appendix I) : 
"crucial", "important", "questionable", " and "not relevant." 

JOB RELEVANCE METHODOLOGY 

The relevance of the content of the NTE Early Childhood 
Education test to the job of a beginning Teacher of Early 
Childhood Education in New York City public schools was assessed 
by the percentage of test items classified as job relevant . Each 
item was classified as either job relevant or iob irrelevant 
based upon the majority of panel member judgments for that item. 



22 



ERLC 



33 



The percentage of job relevant items is considered the primary 
evidence of job relatedness of the whole test. 

In accordance with Educational Testing Service (1985c, pages 
V. 16-17), the relationship of the test to the job of a beginning 
Early Childhood Education may be summarized and described as 
follows: very relevant (percent of questions judged relevant 
greater than or equal to 90), relevant (percent of questions 
judged relevant less than 90 but greater than or equal to 80), of 
questionable relevance (percent of questions judged relevant less 
than 80 but greater than or equal to 65), and not relevant 
(percent of questions judged relevant less than 65). 

In accordance with previous studies (Educational Testing 
Service, 1933; 1985c), two methods were used to assign test 
questions to the job relevant and job irrelevant categories. In 
both Methods I and II, to determine the relevance of individual 
test items, panel member responses to the machine scannable Job 
Relevance Form categories of both "crucial" and "important" were 
classified as job relevant . Furthermore, responses to the "not 
relevant" category were treated as job irrelevant . The 
difference between Methods I and II is the manner in which panel 
member responses to the Job Relevanca Form category of 
"questionable" is used to categorize individual test items as job 
relevant or job irrelevant . 
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In Method I, to determine item job relevance, panel member 
responses to the "questionable" response category were treated as 
job irrelevant responses in the same manner as "not relevant." 
In Method II, however, when determining item job relevance, panel 
member responses to the "questionable" response category were 
distributed proportionately between job relevant and jiob 
irrelevant. 



Method II was implemented as an alternate way of analyzing 
the data because some uncertainty exists as to why panel members 
chose the "questionable" category (Educational Testing Service, 
1985c, p. III. 25-26). Panel members could interpret 
"questionable" in at least the following four different ways: 



(a) The content of the question was deemed to be 
of questionable relevance to the job of the 
teacher (the intended meaning) . 

(b) The content of the question was deemed job 
relevant for only a subgroup of all 
teacher/practitioners, and, therefore, could 
not be considered "Crucial" or "Important" 
for all. 

(c) One or more of the response options to the 
test question could be interpreted in several 
ways. 

(d) The panel member had no opinion as to the 
degree of job relevance of the content of the 
question (a "Do Not Know" option was not 
included for this panel) . 
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VI. RESULTS FROM THE JOB RELEVANCE PANEL 



As indicated earlier, Method I is the more conservative of 
the two methods used to statistically analyze the Job Relevance 
Panel data. Using Method I, 144 items of 150 (i.e., 96.0%) were 
found to be lob relevant . The six items judged iob irrelevant 
were items numbered 11, 12, 30, 32, 84 and 134. 

As indicated earlier, also, Method TI was used to account 
for possible ambiguities in the Job Relevance Panel responses. 
Using Method II, 149 items of 150 (i.e., 99.3%) were found to be 
job relevant . Only item number 11 was found to be lob 

Previous studies (e.g., Educational Testing Service, 1985c, 
page V.15 - V.16) conducted supplementary Chi-square analyses to 
determine if the questions judged to be job irrelevant were 
distributed throughout the test or clustered in particular 
content topics. This was done, however, only for tests for which 
less than 90 percent of the items were judged to be lob 
irrelevant . In the current study, more than 90 percent of the 
items were judged to be 1ob relevant . Furthermore, review of 
Table 7 indicates that the irrelevant items are not clustered in 
any one test content topic. Therefore, Chx-square analyses for 
item clusters were not required. 
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TABLE 7 

Job Relevance Judgment Analysis 
By Test Content Topic* 



METHOD I METHOD II 



Content 
Topic 


Number 

of 
Items 


No. Of 
Irrelevant 
Items 


Percent of 
Relevant 
Items 


No. Of 
Irrelevant 
Items 


Percent of 
Relevant 
Items 


I 


46 


2 


95.7 


0 


100.0 


II 


15 


2 


86.7 


0 


10C.0 


III 


18 


2 


88.9 


1 


94.4 


IV 


43 


0 


100.0 


0 


100.0 


V 


19 


0 


100.0 


c 


100. 0 


VI 


9 


0 


100.0 


r. 


100.0 



*For a more detailed explanation of NTE Early Childhood Education 
test content topics, refer to Table 1. 
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CONSISTENCY OF JOB RELEVANCE RESULTS 

The consistency of judgments about the job relevance of the 
test questions was assessed by comparing the extent to which the 
two half panels agreed. Description of the manner in which the 
half panels were established was presented earlier. The results 
from the two Job Relevance half panels are depicted in Table 8. 

For Methods I and II, a substantial proportion of the items 
were judged job relevant by both half panels. These findings are 
consistent with results obtained for the full panel. For Method 

I, 96.0% and 97.3% of the items were judged as lob relevant by 
Half Panel 1 and Half Panel 2, respectively. Using Method I, the 
full panel judged 96.0% of the items as 1ob relevant . For Method 

II, Half Panels 1 and 2 both judged 99.3% of the items as job 
relevant . This finding was identical to that obtained for the 
full panel. 

Most of the specific items judged iob irrelevant by the full 
panel were judged iob irrelevant by the half panels for Methods I 
and II. Specifically, for Method I, four of the six items judged 
iob irrelevant by the full panel were judged iob irrelevant by 
both of the half panels. For Method II, the one item found job 
irrelevant by the full panel was also judged iob irrelevant by 
both of the half panels. 
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TABLE 8 



Job Relevance Judgment 
Analysis By Half Panels 



METHOD I 



Item Nos. of Percent of 
Irrelevant Relevant 
Items 



Items 



METHOD II 



Item Nos. of Percent of 
irrelevant Relevant 
Items Items 



Half 
Panel 1 



11,12,32, 
84,115,134 



96.0 



11 



99.3 



Half 
Panel 2 



11,12,32, 
134 



97.3 



11 



99.3 



Full 
Panel 



11,12,30, 
32,84,134 



96.0 



11 



99.3 
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DISTRIBUTION OF PANEL MEMBER JUDGMENTS 

The item level data were analyzed, also, for each of the 
panel members who provided judgments, in a manner similar to that 
reported in Educational Testing Service (1987a).* In this Method 
1 analysis, responses of "crucial" or "important" were considered 
job relevant responses. Responses of "questionable" or "not 
relevant" were considered iob irrelevant . Each item was 
considered either as iob relevant or 1ob irrelevant for each 
panel member. The percentage of iob relevant items was then 
calculated for each panel member. These steps were repeated for 
Method II. A distribution of the panel member percentages was 
obtained and ranked. Table 9 presents the maximum, minimum and 
quartile points of the distribution for both Methods I and II. 



TABLE 9 

Distribution of Percentage of Items 
Judged Job Relevant by the Job Relevance Panel 







Maximum 


75 %ile 


Median 


25 %ile 


Minimum 


Method 


I 


98.7% 


92.7% 


86.0% 


80.0% 


66.4% 


Met!-cd 


II 


100.0% 


99.3% 


98.0% 


96.6% 


82.6% 



*These procedures are optional in the sense that they are not 
required by the Guidelines for Proper Use of NTE Tests 
(Educational Testing Service, 1985a). 
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Review of Table 9 indicates that, using Method I, the 
highest ranked person identified 98.7% of the items in the NTE 
Early Childhood Examination test as iob relevant , in contrast, 
the lowe.. z ranked person judged 66.4% of the items as iob 
relevant. The person at the median or fiftieth percentile 
judged 86.0% of the items as iob relevant . The corresponding 
percentages for Method II are also presented in Table 9. 

For both Methods I and II, the ranges of the percentages of 
items judged iob relevant reflected the fact that individual 
panel members judged most items to be iob relevant . For both 
methods, the distributions of percentages were towards the higher 
end of he percentage scale. In fact, the lowest percentage was 
t^e Method I minimum of 66.4%. The range for Method I (i.e., 
32.3) was greater than the range for Method II (i.e., 17.4). The 
percentages derived from Method II were higher on the percentage 
scale than those from Method I. Distributions from both Methods 
I and i:: were negatxvely skewed. In other words, both 
distritut: ons bunch towards the upper ends of their respective 
ranges, with relatively few '- ores towards the lower ends. 
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VII. DISCUSSION OF JOB RELEVANCE PANEL RESULTS 



It appears that the NTE Early Childhood Education test may 
be characterized as "Very Relevant" to the job of beginning 
Teacher of Early Childhood Education in New York City schools. 
To determine the meaningfulness of the results of the Job 
Relevance Analysis, two considerations are noted. 

First, previous studies (i.e., Educational Testing Service, 
1985c, pages V. 16-17) indicate that the relevance of the NTE 
tests to the job of beginning teacher may be described as "Very 
Relevant" if the percent of questions judged job relevant is 
greater than or equal to 90%. In the current study, both Methods 
I and II yielded relevant judgments greater than 90%. 
Consequently, the relevance of the NTE Specialty Area Test in 
Early Childhood Education to the job of beginning Teacher of 
Early Childhood Education in New York City public schools may be 
described as "Very Relevant." 

Second, the NTE Early Childhood Education test has been 
judged as having a percent of 1ob relevant itemr among the 
highest of all NTE Specialty Area Tests previously validated for 
New York State. Specifically, using Method I, the percentage of 
job relevant items for other NTE Specialty Area Tests has ranged 
from 79.2% to 96.7% as demonstrated in other validity studies 
(Educational Testing Service, 1985c, Table III. 14, p. 36). As 



indicated above, in the current study, Method I resulted in 96.0% 
of the items judged iob relevant . 

Using Method II , the percentage of iob relevant items for 
other NTE Specialty Area Tests has ranged from 91,3% to 100.0% as 
demonstrated in other validity studies (Educational Testing 
Service, 1985c, Table III.19A, p. 49), As indicated earlier, in 
the current study, Method II resulted in 99.3% of the items 
judged iob relevant. 

Consistency of the percentage of items judged iob relevant 
by the Job Relevance Panel was demonstrated by assessment of 
results based upon half panels. There is a substantial 
consistency of percentage of items ji *.ged iob relevant by the 
half panels both with each other and with the full panel. Most 
of the specific items judged either iob relevant or iob 
irrelevant were consistently judged so by both of the ha^f panels 
and by the full panel. 
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VIII. KNOWLEDGE ESTIMATION 



The Knowledge Estimation Panel was established to provide 
data which serves as the basis for identifying a minimum 
qualifying score or passmark. This passmark will serve as the 
minimum qualifying score which estimates the minimum level of 
knowledge and skills required for competent performance by entry- 
level New York City Teachers of Early Childhood Education. The 
Knowledge Estimation Panel was comprised of 32 members including 
those 31 individuals who had served previously on the Job 
Relevance Panel. 

KNOWLEDGE ESTIMATION PA NEL PROCEDURES 

When the Knowledge Estimation Panel met, the tasks to be 
performed were discussed. Subsequently, panel members were asked 
to read the "overview of Tasks.. ." (see Appendix L) and 
"Instructions for Completing the Knowledge Estimation Form" (see 
Appendix M) • In addition,, panel members were presented the 
machine-scannable "Ki owl edge Estimation Form 11 (see Appendix N) 
and a Panel*st»s Comment Sheet (see Appendix J). Next, panel 
members were presented with a brief exercise in standard-setting 
(see Appendix 0) . 

Knowledge Estimation Panel members estimated the proportion 
of minimally knowledgeable e.itry-level teachers who would know 
the answer to each test question, m making their estimates, 
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panel members were instructed to assume that the b^jinning 
teachers would not guess blindly if they did not know the answer. 
Furthermore, panel members were instructed to estimate the 
percentage who would know the answer without considering the 
possibility that some additional teachers might pick the answer 
purely as a lucky guess (see Appendix M for "Instructions for 
Completing the Knowledge Estimation Form") • To provide their 
estimates, Knowledge Estimation Panel members used an eight- 
category scale (see Appendix N) , as follows: "2%", "10%", "25%", 
"40%", "60%", "80%", "98%", and "DNK" (Do Not Know). 

KNOWLEDGE ESTIMATION METHODOLOGY 

The abovementioned instructions were given to the panel 
because candidates are discouraged from guessing blindly when 
they are administered this NTE Specialty Area Test. It is noted 
that the NTE Bulletin of Information (Educational Testing 
Service, 1987c, page 21) states that: "Your score on a Specialty 
Area Test is based on the number of scored questions you answer 
correctly minus a fraction of the number you answer incorrectly. 
You should know that you will lose one-fourth of a point for each 
fi/e-choice question you answer incorrectly." This "formula- 
scoring" encompasses a correction for guessing. Therefore, the 
panel judgment data collected and statistically analyzed in this 
study do not require a correction for guessing. 



34 



Data from the Knowledge Estimation Panel judgments were 
included only for those items previously characterized as job 
relevant , based upon judgments by the Job Relevance Panel, 
Calculations were based upon aggregates of both 144 and 149 items 
identified as job relevant by Method I and II , respectively , of 
the Job Relevance Panel statistical analyses. The study-derived 
minimum passing average or mean raw score was estimated by 
calculating a weighted mean of "item percentage" judgments 
(converted to proportions) from the 32 Knowledge Estimation Panel 
members to all iob relevant items in this test. The weighted 
mean was required because 18 invalid multiple responses were 
eliminated from the total of 4,800 possible responses. 

More specifically, to derive the mean raw score for Method 
I, a score was obtained for each panel member by summing that 
panel member's responses across the 144 iob relevant items. This 
process was repeated for Method II, where 149 items were 
included. Whers there were invalid (i.e., missing or multiple) 
responses, the sum was increased by a factor proportional to the 
number of valid responses (i.e., 144 divided by the number of 
valid responses) provided by that panel member. Subsequently, 
the scores of each of the 32 panel members were summed and 
divided by 32. The result is the estip^ted mean raw score 
expected of minimally knowledgeable beginning teachers by the 
panel of experts. Subsequently, this score is converted to a 
mean scaled score using an NTE score conversion table and 
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equation provided by the Educational Testing Service (1988a) . 
This mean scaled score is the frame of reference for setting the 
passmark. 



This mean scaled score is rounded to the nearest "10 " to 
match the manner in w\±ch the Educational Testing Service reports 
individual scores. This is in accordance with Educational 
Testing Service (1985c, pages IV. 3 and IV. 4 and Table IV. 1 on 
page 60) which states: "...state officials are advised to select 
as the standards for certification three-digit scores that end in 
zero." Consequently, the rounded mean scaled scores are three 
digit scores that end in zero. 

Ratings were assessed for each of the 32 Knowledge 
Estimation Panel members in order to review the distribution of 
scores. The rating of each of the 32 panel members considered 
separately was obtained by summing the response category 
percentages (converted to proportions) selected by each panel 
member for all iob relev. nt items for each method (i.e., 144 for 
Method I ind 149 for Method II) . 

Knowledge Estimation Panel member judgments were reviewed, 
also, for each of the iob relevant items. The mean difficulty of 
each item considered separately was obtained by summing the 
response category percentages (converted to proportions) selected 
by the 32 panel members for each item and dividing by the total 
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number of judgments (i.e., usually 32) for that particular item. 
Subsequently, the mean item difficulties were aggregated to 
obtain the mean raw score for each of the six Content Topics of 
the NTE Specialty Area Test in Early Childhood Education. 

The consistency of panel judgments was assessed by 
demonstrating the degree of similarity between the judgments of 
the half panels. The manner in which the half panels was 
established was discussed earlier in the section entitled: 
"Establishing the Half Panels." 
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IX. RESULTS FROM THE KNOWLEDGE ESTIMATION PANEL 



Results for both Method I and Method II were obtained, as 
both raw scores and converted or scaled scores, for the full 
panel. In addition, results for Knowledge Estimation half panels 
were statistically analyzed to estimate the consistency or 
reliability of results. 

For the full Knowledge Estimation Panel, a minimum passing 
average or mean raw score and corresponding scaled score were 
obtained for both Methods I and II. As indicated in Table 10, 
for Method I (144 items) the mean raw score was 64.2 (with a 
standard deviation of 19.6). For Method II (149 items), the mean 
was 66.4 (with a standard deviation of 20*3). 

Table 10 presents, also, the mean difficulty levels for the 
entire test and for each of the six NTE Early Childhood Education 
Test Content Topics for both Methods I and II. Overall, there 
appears to be no meaningful difference between Methods I and II 
for the full test. As indicated earlier in Table 7, two items 
were eliminated as lob irrelevant from each of Content Topics J, 
II and III. No items were deleted as iob irrelevant from Content 
Topics IV, V and VI. Consequently, Table 10 reveals that there 
are no differences between Methods I and II for Content Topics 
IV, V and VI. 
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TABLE 10 



Knowledge Estimation Judgment Analysis 
By Test Content TopLc* 







Method 


I 




Method 


II 


Content 
Topic 


No. 
of 
Items 


Mean 


Standard 
Deviation 


NO. 
Of 
Items 


Mean 


Standard 
Deviation 


I 


44 


18.9 


6.1 


46 


19.7 


6.4 


II 


13 


6.0 


1.7 


15 


7.1 


2.0 


III 


16 


5.9 


2.7 


17 


6.1 


2.9 


IV 


43 


20.4 


6.1 


43 


20.4 


6.1 


V 


19 


8.8 


2.7 


19 


8.8 


2.7 


VI 


9 


4.1 


1.5 


9 


4.1 


1.5 


Test 


144 


64.2 


19.6 


149 


66.4 


20.3 



*For a more detailed explanation of NTE Early Childhood Education 
test content topics, refer to Table 1. 



CONVERTED SCORE RESULTS 

To be consistent with the scale on which the NTE Specialty 
Area Test in Early Childhood Education is reported, the mean raw 
scores derived from Knowledge Estimation Panel judgments were 
converted using a table and equation provided for this purpose by 
the Educational Testing Service (1988a). The converted or scaled 
scores for the NTE Early Childhood Education test are obtained by 
multiplying the mean raw score by 3.9016 and adding a constant of 
254. 1795. Subsequently, the converted scores are rounded to the 
nearest multiple of ten. For this equation, the possible scaled 
score range for the Early Childhood Education test with all 150 
items is 250 to 840. Applying these steps to the mean raw scores 
resulted in scaled scores of 504.5 (with a standard deviation of 
76.4) for Method I and 513.1 (with a standard deviation of 79.2) 
for Method II. The corresponding rounded scaled scores are 500 
and 510, for Methods I and II, respectively. 

As indicated earlier, ratings were assessed for each of the 
32 Knowledge Estimation Panel members in order to review the 
distribution of scores. Table 11 presents the raw and scaled 
score points corresponding to various positions in the 
distribution of Knowledge Estimation Panel judgments of the score 
a minimally qualified beginning teacher would receive (see, for 
distributions of three other NTE Specialty Area Tests, 
Educational Testing Service, 1987a, p. 27). 
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TABLE 11 

Distribution of Knowledge Estimation Panel Judgments 





Mean of 
Panel 


Distribution of Panel Judaments 




Judgments 


Max. 


75 %ile 


Median 


25 %ile Min. 


Method I 












Raw Score 


64 . 2 


111.4 


77.5 


61. 6 


51.0 28.8 


Scaled Score 


504.5 


688.8 


556. 5 


494.3 


453.1 366.4 


Method II 












Raw Score 


66.4 


114.8 


80. 0 


64. 1 


52.7 29.8 


Scaled Score 


513. 1 


702.0 


566. 4 


504. 1 


460.0 370.4 



9 

ERIC 



41 

52 



In Table 11, the minimum and maximum, respectively, 
represent the lowest and highest ratings made by some panel 
member to all iob relevant items. Specifically, Table 11 
indicates that the scaled scores ranged from 366.4 to 688.8 for 
Method I and 370.4 to 702.0 for Method II. The column headed 
••Median" presents the scores of the panel member at the fiftieth 
percentile. Review of Table 11 indicates that the median was 
494.3 for Method I and 504.1 for Method II. in addition, the 
columns headed 75 %ile and 25 %ile, respectively, present the 
scores of the panel members who were at the 75th and 25th 
percentiles. It is noted that the distributic from both 
Methods I and II were positively skewed. In other words, both 
distributions bunch towards the lower ends of their respective 
ranges, with fewer scores at the higher ends. 

Compared with results of other NTE Specialty Area Tests, the 
rounded mean scaled scores obtained in this study for the Early 
Childhood Education test appear relatively low. However, 
although scores on the various NTE Specialty Area Tests appear 
comparable, scores on the different tests should not be compared 
(Educational Testing Service, 1987c, page 22). For informational 
purposes, it is noted that the rounded mean scaled scores based 
upon the judgments of New York City educators were 500 for Method 
I and 510 for Method II. Other validity studies conducted by the 
New York State Education Department for 16 other NTE Specialty 
Area Tests have found the scaled scores to fall within the ranges 
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of 550 to 682 for Method I, and 566 to 714 for Method II (see 
Educational Testing Service, 1985c, Table V.9, page 74). 

CONSISTENCY OF KNOWLEDGE ESTIMATION RESULTS 

Summary statistics of the judgments for each of the two half 
panels for Methods I and II are presented in Table 12. For each 
method, comparison of the size of the difference in judgments 
between the half panels reflects the degree of consistency. The 
mean raw score difference between half panels is approximately .9 
for Method I and 1.2 for Method II. 

When compared with results from other studies for other NTE 
Specialty Area Tests (see Educational Testing Service, 1985c, 
pages III. 30 and III. 31), results from this study's Knowledge 
Estimation Panel appear to be more consistent. Specifically, for 
Method I the scaled score difference between the half panels was 
approximately 3.8. Other studies report corresponding 
differences for other NTjc! Specialty Area Tests as ranging from 4 
to 72. Similarly, for Method II the difference was approximately , 
4.7 compared with a range of 6 to 72 reported elsewhere. 

In addition to the analyses for consistency at the test 
level , degree of consistency at the item level was analyzed 
in accordance with the analysis conducted in Educational Testing 
Service (1985c, page III. 31). This involved finding the item 
means for all the job relevant items for each half panel. 
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TABLE 12 



Knowledge Estimation Panel Judgments 
By half Panels 





METHOD I 


METHOD II 




Mean 


Standard 
Deviation 


Mean 


Standard 

Deviation 


Half Panel 1 










Raw Score 


64.6 


20.9 


67.0 


21.4 


Scaled Score 


506.4 


81.4 


515.5 


83.7 


Half Panel 2 










Raw Score 


63.7 


18.9 


65.8 


19.8 


Scaled Score 


502.6 


73.8 


510.8 


77.2 
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The item mean is calculated by summing the half panel responses 
for an item and dividing the derived sum by t\e number of valid 
responses. Subsequently, the absolute values of the differences 
between the corresponding item means of the half panels were 
summed. This resulted in values 7.611 for Method I (144 items) 
and 7.923 for Method II (149 items). The values of this 
statistic have ranged from 6 to 16, for 16 other NTE Specialty 
Area Tests in studies conducted for the New York State Education 
Department (Educational Testing Service, 1385c, page III. 32). 

For further evidence of consistency, the degree of relative 
correspondence between the estimates of the half panels for 
individual test items, as represented by the correlation between 
them, was examined. This resulted in a correlation coefficient 
of .7952 for 144 items and .7766 for 149 items. The values of 
these correlation coefficients have ranged from .37 to .90 for 16 
other NTE Specialty Area Tests in studies conducted for the New 
York State Education Department ; Educational Testing Service, 
1985c, page III. 32) . 

To the extent that t- half panels are equivalent, the 
correlation between their judgments can be interpreted as half- 
length reliability coefficients. The reliability of the 
percentage judgments for the full panel can be estimated by 
application of the Spearman Brown Formula (Educational Testing 
Service, 1985c, page III. 32). 
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Comparability of half panel member teacning experience is 
demonstrated by data presented in Table 6. Applying the 
abovementioned correlation coefficients, the Spearman-Brown full 
test reliability estimates are .88592 for Method I and .87425 for 
Method II. These indices provide evidence of the reliability of 
Knowledge Estimation Panel judgments. It is noted, also, that 
Tables 8 and 12 depict the similarity of half panel member 
judgments. For informational purposes, it is noted that 
Spearman-Brown reliability estimates have ranged from .59 to .94 
for 16 other NTE Specialty Area Tests in studies conducted for 
the New York State Education Department (see, Educational Testing 
Service, 1985c, page III. 32). 
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X. SETTING THF PASSMARK 



Based upon expert judgments provided in this study by the 
panels of New York City professional educators, and additional 
considerations specified below, the recommended passing score for 
the NTE Specialty Area Test in Early Childhood Education is 500. 
This score can be considered an estimate of the mini aum entry- 
level performance expected of new Teachers of Early Childhood 
Education in New York City public schools. 

The Method I rounded mean scaled score of 500, rather than 
the score of 510 derived from Method II, is recommended, in part, 
because the NTE Core Battery passmarks established by the New 
York State Education Depa cment were based upon Method I 
analyses. It is anticipated that the actual decision on which 
specific passmark to select will be made by New York City Board 
or Education policy-makers. 

Policy-makers may decide, however, to modify the 
recommended passmark based upon various considerations. For 
example, when establishing passmarks, the standard error of 
measurement (SEM) must be taken into consideration. Previous 
studies by the Educational Testing Service have reported results 
in tables which include the mean and additional scores which are 
one and two SEMs below and above the mean. Consequently, Table 
13a presents these scores for Methods I and II, respectively. 
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Table 13b presents the corresponding rounded scaled scores. The 
"rounding 11 is the last step in establishing passmarks. Any of 
the scores in Table 13b might be potential passmarks, under 
certain circumstances. 

In a statistical sense, compared with other scores presented 
in Table 13b, the mean scores are most reflective of the 
estimates obtained from Knowledge Estimation Panel members in 
this study. If passmarks which are one or two SEMs below the 
rounded mean scaled scores are instituted, however, there is less 
risk of incorrectly rejecting potentially qualified candidates 
than if the mean score had been instituted as the passmark. In 
effect, using a lower passmark results in a greater proportion of 
potentially eligible teachers being selected than would occur if 
higher passmarks were used. Consequently, a relatively large 
select -on pool of potential teaci^rs can be established. This 
may be necessary to meet anticipated personnel needs. 

It is important to note, however, that by establishing lower 
passmarks, there is an increased risk of accepting potentially 
unqualified candidates. Therefore, the risks of certifying 
unqualified applicants compared to the risks of noc certifying 
qualified applicants must be carefully considered (see, for 
discussion, Educational Testing Service, 1985c, chapter IV) • 




48 



S3 



TABLE 13 



Knowledge Estimation Panel Judgments and 
Standard Error of Measurement 



Table 13a. Calculated Scaled Scores 



Below Mean Above Mean 

-2 SEM* -1 SEM Mean +1 SEM +2 SEM 

Method I 450.5 477.5 504.5 531. 5 558.5 

Method II 459.1 486.1 513.1 540.1 567.1 



Table 13b. Corresponding Rounded Scaled Scores 



Below Mean Above Mean 

-2 SEM -1 SEK Mean +1 SEM +2 SEM 

Method I 450 480 500 530 560 

Method II 460 490 510 540 570 



♦"SEM" means "Standard Error of Measurement." In this 
Table the "Standard Error of Measurement" is 27, 
based upon Educational Testing Service , 1988b. 
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In addition to taking into account the Standard Error of 
Measurement for the Early Childhood Education test, other factors 
must be considered when establishing passmarks. These factors 
may include, for example, the: need for adequate numbers of 
qualified new teachers, supply of new teachers, need to maintain 
an adequate pool of teachers from all segments of the New York 
City population, and existence of other factors as prerec sites 
for licensure. 

To provide a frame of reference, nationwide NTE Early 
Childhood Education test performance data are presented in Table 
14. More detailed test score distributions can be found in 
Educational Testing Service (1986, 1987d, 1988b) . Review of 
Table 14 indicates that the rounded mean scaled scores of 500 and 
510, respectively, correspond to percentiles of 16 and 17 for the 
1985-88 nationwide test score distribution (Educational Testing 
Service, 1988b) . In other words, the rounded mean scaled scores 
of 500 and 510, respectively, are exceeded by 84 and 83 percent 
of the candidates nationwide. Consequently, if either of these 
scores i*are adopted, without modification, a relatively large 
potential selection pool of candidates may be established. 
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TABLE 14 



Nationwide Test Data For The NTE Specialty Area Test 
in Early Childhood Education* 





1983-86 


1984-87 


1985-88 


Mean Scaled Score 


586 


591 


599 


Standard 
Deviation 


102 


100 


98 


Number of 
Examinees 


11,643 


12,379 


13,152 


Percentile 
Corresponding 
To score of 500 


19 


18 


16 


Percentile 
Corresponding 
To score of 510 


21 


20 


17 



*Examinees were Seniors or held a Bachelor's degree and were tested within 
three year intervals. For example, the column headed "1983-86" includes test 
scores obtained within the interval beginning October, 1983 and ending July, 
1986. Sources of test data are: Educational Testing Service (1986; 1987d; 
1988b). 
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In conclusion, federal "Uniform Guidelines" (Equal 
Employment Opportunity Commission, et al. , 1978, Section 5.H., 
page 38298) indicate that: "where cutoff scores [i.e., 
passmarks] are used, they should normally be set so as to be 
reasonaLle and consistent with normal expectations of acceptable 
proficiency within the work force." This study has provided 
information which is necessary for establishing such cutoff 
scores for entry-level New York City Teachers of Early Childhood 
Education. 
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Appendix A 



BOARD OF EDUCATION 




OP THE CITY OP NEW YORK 



HO UVINGSTON STREET 



BROOKLYN, N Y U20I 



OOLORES M. FERNANDEZ, Pm 0 



MEMORANDUM 



oc»ut* Chan iellor 
for instruction and development 



May 19, 1989 



RE: 



TO: 



FROM 



: Dolores M. Fernandez /) //} — — 7~~ 

NTE Specialty Area Test in Early Chjddhood Education 



COMMUNITY DISTRICT SUPERINTENDENTS 




I would appreciate your assistance in rrranging for the 
participation of one of your staff in a one-day validity study of 
the NTE Specialty Area Test in Early Childhood Education. 

Specifically , I would appreciate your assigning either your 
Early Childhood Education representative or some other designee 
who is particularly knowledgeable of our Early Childhood 
Education curriculum and instruction. We are including one such 
reprasentative from each Community School District. 

This individual will participate in a professional panel 
which will meet on Friday, June 16, 1989 at 9:00 a.m. in room 525 
of 110 Livingston Street, Brooklyn, New York. The session will 
last approximately six hours including a lunch break. 

This study is absolutely essential for New York City to 
comply with NTE Guidelines, without this validity study, we have 
been apprised that we will not be able to use the NTE Early 
Childhood Education test to hire teachers. As you know, we have 
been using the NTE tests to hire teachers under the alternate 
route provided for by decentralization law 2590-j. 

Would you please call the Project Director, Dr. Gary M. 
Kippel, at (718) 935-2303 to inform him of your nominee and to 
ask any questions which may arise. I have appended a sample 
letter which you might consider sending to the nominee. Please 
note that the nomine uust contact the Project Director 
immediately. 

Thank you, in advance, for considering this request. 



DMF:cw 
Enclosure 
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Appendix A - Continued 



SAMPLE LETTER FROM COMMUNITY SUPERINTENDENT 
TO PANEL MEMBER NOMINEE 



Dear (Nominee) , 

I would like to invite you to represent our district in a 
very important study. 

Specifically, you are bexng asked to serve on a one-day 
panel which is part of a New York City school system validity 
study of the NTE Specialty Area Test in Early Childhood 
Education. This panel will meet on ( date ) 1989 at 

( time ) a.m. at ( location) . The session will last 

approximately si.: hours including a lunch break. 

You are being invited to represent our district because you 
are recognized as an expert in Early Childhood Education. I hope 
your schedule will permit you to participate in this project. 
You will be asked to make judgments about the relevance of this 
test to the knowledge needed by a beginning teacher. In 
addition, you will be asked to estimate test performance required 
of minimally knowledgeable applicants for certification. 

If you are able to participate, please call the Project 
Director, Dr. Gary M. Kippel, at (718) 935-2303, He will: answer 
any questions which may arise, send you the necessary forms and 
keep me apprised of your participation. 

If you are unable to participate, please immediately call 
( name ) in my office so that we may recommend an alternate 
panel member to represent our school district. 

Thank you, in advance, for considering this request. 



Sincerely, 



Community Superintendent 



GMK:cw 
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Appendix B 
BACKGROUND INFORMATION SHEET 



1. Name: 



Position / Title: 



2. Social Security Number 



3. Addresses and Telephone Numbers 



Home 



School/District^ 
No., Street 



City, State Zip 

Telephone NO. ( ). 



( >. 



4. Professional Experience 



Total Years of Teaching Experience 



Employer (last 5 years) Position (Department, Subject) 



Dates 



A) Male Female 

B) Highest degree: 



5. Demographics 



Bachelors Masters Doctorate 



C) Age Range: 

D) (OPTIONAL) 



Major : 



.18-25, 26-32, _33-40, _41-48, _49-56, 57 and over 

" White, Black, Hispanic , American Ind: , 



Alaskan Native , Pacific Islander, Asian, Other 



6. Certification/Licenses Held 

Subject /Discipline 

A. Board o* Examiners Regular License 

National Teacher Examination 

Regular Temporary Per Diem Certificate (TPD) 

TPD Via New York State Certification 



B. Form: Provisional, 



Permanent , 
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OTHER ( TXPLAIN ) 



Appendix C 



PERMISSION TO BE LISTED AS A PARTICIPANT 
IN THE VALIDITY STUDY OF THE 

NTE SPECIALTY AREA TEST 
IN EARLY CHILDHOOD EDUCATION 



I AGREE TO HAVE MY NAME LISTED AS A PANELIST IN THE 
FINAL STUDY REPORT. 



I DO NOT WISH TO HAVE MY NAME LISTED AS A PANELIST 
IN THE FINAL STUDY REPORT. 



(Signature) 



NAME (Please Print) 
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Appendix D 



VALIDATION OF THE NTE SPECIALTY AR£<\ TEST IN 
EARLY CHILDHOOD EDUCATION 

PROFESSIONAL PANEL MELTING 

June 16, 1989 

. m 9 A.M. 

110 Livingston Street, Room 525 
Brooklyn, NY 11201 

AGENDA 

Registration and Coffee 

Welcome and Introductions 

Discuss Agenda and Objectives 
Complete Non-Disclosure/Confidentiality Form 
Complete Background Information Sheet 
Complete "Permission To Be Listed..." Form 

Orientation of Job Relevance Panel 
Discuss and Read Test Content Description 
Distribute NTE Test Booklets 

Discuss and Read: Overview of Tasks To Be Prepared 

by the Job Relevance Panel 
Discuss and Read: Instructions for Completing the Job 

Relevance Form 
Discuss: Machine Scannable Job Relevance Form 
Discuss: Panelist's Comment Sheet 
Discuss: Test Comprehensiveness Comment Sheet 
Answer Panel Member Questions 

Job Relevance Panel Provides Judgments of Questions 
Job Relevance Panel Completes Comment Sheets 
Discussion 

Break 

Orientation of Knowledge Estimation Panel 

Discussion of Standard-Setting Issues 

Discuss and Read: Overview of Tasks to be Performed 

by Members of the Knowledge Estimation Panel 
Discuss and Read: Instruction for Completing the 

Knowledge Estimation Form 
Standard-Setting Exercise: Examinee Knowledge... 
Discuss: Panelist's Comment Sheet 
Answer Panel Member Questions 

Knowledge Estimation Panel Provides Judgments 
Knowledge Estimation Panel Completes Comment Sheet 
Discussion 
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Appendix E 



test name Early Childhood Education 



form 



NTE DISCLOSURE POLICY and RELEASE FORM 



I understand the importance of protecting the security of 
the NTE Core Battery and Specialty Area tests, I accept 
responsibility for the proper safeguarding of these confidential 
tests and agree to the following conditions: 



1. All copies of the tests will remain in full view of the 
ETS /^presentative during thr entire inspection period, 

2. My copy of the test will be returned to the ETS agent 
each time I leave the meeting room, and at the end of 
the inspection period. 

3. I will not take an NTE test for a period of one year 
following the workshop without requesting permission 
from ETS in writing at least six weeks prior to a 
scheduled test date. 

4. i will not take notes, or otherwise record, copy, or 
disclose items or responses during or after the 
workshop. 

5. If for any reason the ETS representative must leave the 
meeting room, all test copies will be gathered and 
returned to the ETS agent. 



signed 

name (please print) 



title 



institution/agency New York City Board of Education 
. address 
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TEST CONTENT DESCRIPTION 



Introductory Statement 
NTE Early Childhood Education Specifications 



The Early Childhood Education Examination is based on a teaching approach 
that emphasizes the involv MDent of young children In a variety of 
child-centered activities. Flay is used to encourage children's active 
Involvement Jn learning. Activities should allow children opportunities for 
choices, decision-making, and discovery. The curriculum should reflect a 
concern for the development of the whole child, Including the child's 
physical, cognitive, social, and langoage development. 

Practices should be based on proven research and should involve the 
family, school, and conrounity as resources. Practitioners in this field 
shield be knowledgeable about auch things as the effects of positive 
self-concept, multicultural influences, and racial/sexual discrimination on 
children's development. A multisensory approach to instruction is advocated 
with a particular emphasis on concrete experiences. Early childhood teachers 
should be able to understand currlculin and developmental theory and be able 
to apply that knowledge in the early thlldhoud education setting. 

Insofar as possible, questions for this examination are focused on 
testing the examinee's understanding and recognition of appropriate 
applications of early childhood education knowledge and theory. The majority 
of questions are based on children ages 3 through 8 although some questions 
may require knowledge of development at earlier and later ages to test the 
examinee's understanding of the full developmental range within which children 
in early childhood education settings can be found. 



62 



Appendix F - Continued 
-2- 



OVERVIEW OF TEST CONTENT 

EARLY CHILDHOOD EDUCATION 

The examination in Early Childhood Education is intended primarily 
for candidates who have completed their undergraduate preparation in 
the field. In keeping with trends in the field, the examination 
focuses on the candidates' understanding and recognition of 
appropriate applications of early childhood education knowledge and 
theory related to the development of the whole child. Major 
dimensions of the examination include the following areas: 

APPROXIMATE 

CONTENT TOPICS % OF TEST 

I. The nature of growth, development, and learning of 30% 

young children - including aspects of cognitive 
development (concepts, skills, language), physical 
development, and personal -social development 

II. Factors that influence growth and development - 10% 

including biological, familial, nutritional- 
hygienic, and cultural factors 

III. The contributions of developmental and curriculum 12% 

theory to early childhood education practices - 
including major streams of developmental theory 
(e.g., cognitive, behaviorist, social-learning) 
and major streams of curriculum theory (e.g., Bank 
Street, Bereiter-Englemann, Kamii, Montessori) 

IV. The planning and implementation of curriculum - 29% 

including the planning, selection, and implemen- 
tation of appropriate curriculum experiences; 
management of the physical learning environment; 
behavior management issues and practices; and 
utilization of family-community resources in the 
learning program 

V. The evaluation and reporting of student progress - 13% 

including the selection and use of formal and 
informal assessment instruments or procedures, the 
maintenance of records of a chi 1 d 1 s progress, and 
effective communication with parents about a child's 
total developmental progress 

VI. Professional and legal responsibilities of the early 6% 

childhood teacher - including effective teacher 
interaction with other adults in the learning setting 
and cognizance of legal regulations that impact on 
teaching in the early childhood setting 

Used with copyright permission of the Educational Testing Service. 
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Appendix F - Continued 
-3- 

DETAILS OF TEST CONTENT 

NTE EARLY CHILDHOOD EDUCATION 
Ages 3 through 8 

150 Questions 



_ Approximate 
i of Items 

(46) I. Understanding the Nature of the Growth, Development, and 

Learning of Young Children 



(28) A. Cognitive 

(7) 1. Concept development (e.g., the physical, social, and 
biological world, relevant causal relationships, 
signing in all communication systems (mathematics, 
language, art, music, and drama)* 



(7) 2. Skill development (e.g., manipulative, perceptual, 
symbol recognition and representation, logical 
reasoning, mathematics, reauing] 



(14) 3, Language development 

(7) a. Oral (e.g., listening comprehension, speaking 
(expressing ideas, storytelling, dramatizing, 
bilingual skills), developing vocabulary, 
understanding the systems of language] 



(7) b. Written (e.g., expressing ideas in writing, 

receiving Ideas from print, becoming aware of 
different modes of writing, of inventive and 
conventional spelling, and of conventions of 
writing] 

(9) B. Physical (e.g., typical and atypical growth and 

development, principles of physical development, fine and 
gross motor development, symptoms of illness r health and 
•afetj 

(9) C. Personal-social (e.g., self-concept, learning style, locus 

of control, dependency/Independence, temperament, stages 
of social behavior, gender role, sex, physical 
attractiveness, effects of discrimination, stereotypes, 
aggression] 



Used with copyright permission of the Educational Testing Service. 
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Appendix F - Continued 
-4- 



Approximate 
¥_ of Items 

(15) ii # Recognizing Factors that Influence Growth and Development 

(3) A. Biological (e.g., the effect of genetic maturat ional 

factors] 

(5) B. Familial (e.g., the impact of family relationships % 

parental child-rearing attitudes, ribling relationship, 
birth order] 

(3) C. Nutritional/Hygenic (e.g., the effects of diet and eating 

behaviors, sleep patterns, exercise, immunization, 
environmental conditions] 



(4) D. Cultural (e.g., the effects of the dominant cultural 

values, the role of the primary transmitters of the 
culture (the family, schools, mass media), the effects 
economic, political, and dominant minority cultural 
influences (regional, ethnic, and religious)] 



of 



(18) III. Recognizing Appropriate and Inappropriate Applications of 

Developmental and Curriculum Theory 

(9) a. Understanding the contributions of major streams of 

developmental theory to early childhood education 
practices (e.g., psychoanalytic, cognitive, 
social-learning, behavioxist] 

(9* B. Understanding major streams of early childhood curriculum 

theorv (e.g., Bank Street, Bereiter/Engelmann , Comenius, 
Froebfel, Montessori, Kaoii, Weikert] 
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-5- 



Approxlroate 
# nf Items 



(43) IV. Planning and Implementing Curriculum 

(24) A. Selecting and implementing appropriate curriculum 

experiences 

(4) I. Whole language, language experience, and basal 
approaches 

(<*) 2. Mathematics manipulat Ives 

(4) 3. Inquiry and discovery in science 

(4) 4. Physical/motor experiences 

(4) 5. Aesthetic and affective experiences 

(4) 6. Social experiences 

(5) B. Organizing the physical learning environment [e.g., irrdoor 

and outdoor space, materials anj equipment] 

(3) C. Utilizing outside resources in curriculum planning and 

ircpleaentation Je.g., fatcily, community] 

(9) D. Managing the classroom 

(5) 1. Helping young children learn to manage their own 

behavior 

(4) 2. Recognizing how the teacher's behavior and attitudes 
affect Che learning and development of young children 

(19) V. Evaluating and Reporting Progress 

(7) A. Selecting and using formal and informal assessment 

instruments for evaluating 

(4) 1. developmental progress 

(3) ' 2. effectiveness of curriculum experiences 

(5) B. Maintaining useful records of a child's development and 

progress in learning 

(7) C. Communicating effectively with parents about a child's 

total developmental progress 

(9) VI. Understanding Professional and Legal Responsibilities 

(4) A. Maintaining effective interactions with other adults who 

function within the learning setting 

(5) B. Being cognizant of legal responsibilities and regulations 

that impact on teaching 
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OV 
xEJR 

OVERVIEW OF TASKS TO BE PERFORMED BY MEMBERS OF 
THE JOB RELEVANCE PANEL 

The study in which you have been asked to participate is being 
conducted to determine the validity of the NTE Specialty Area 
Test in Early Childhood Education. One purpose of the study is 
to evaluate the content of this NTE Specialty Area Tests in 
relation to the knowledge and academic skills which are relevant 
to professional practice in the public schools of New York City. 

You have been selected to serve on the Job Relevance Panel. As a 
member of this panel, you will examine individual test questions 
and judge the extent to which the knowledge or academic skills 
needed to answer the question correctly are relevant to competent 
performance as a beginning teacher of Early Childhood Education 
in New York City . 

School personnel from throughout the city are being assembled to 
nuJce their judgments. The judgments, however, will be madr 
individually and independently. Members of the panel will not 
confer as a group, nor will they be informed of the judgments 
made by other members. The judgments of all members of a panel 
will be combined statistically to arrive at a summary judgment 
for the panel about each test question. The summary results for 
the test questions also will be combined, and the final summary 
results will be published in a report describing the study and 
its findings. 

Several items are intended to help you prepare for your tasks. 
If, in studying the materials, you find that you have questions 
about the tasks, be sure they are answered during the orientation 
session. 

1. The Test Content Description identifies the major 
groups of topics covered by the test and indicates the 
relative emphasis given to each topic. This sheet is 
provided to help you become more familiar with the 
general content of the test you are to review: you will 
not be asked to make judgments about the topics and 
relative emphases. However, a Panelist's Comment Sheet 
is provided for your use. 

2. The Job Relevance Form will be used by panelists to 
record their judgments about test questions. Please 
study the instructions for completing this form. 

( continued ) 

Used with copyright permission of the Educational Testing Service. 
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You have been asked to participate in this study because you are 
familiar with the knowledge and academic skills needed for 
competent performance as a beginning teacher of Early Childhood 
Education in New York City. Your qualifications to participate, 
in this study are an important part of the study methodology. In 
order that the final report is as informative as possible, we are 
asking for your permission to identify you in the report. A 
consent form is enclosed and will be collected. In the final 
report, your name will not be associated with your individual 
j udgments . 
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INSTRUCTIONS FOR COMPLETING THE JOB RELEVANCE FORM 

Please print your full name and darken the corresponding ovals. 
The other information is NOT necessary. 

Your task is to make judgments about the extent to which the 
knowledge and academic skills tested by individual questions are 
relevant to competent performance as a beginning teacher in the 
schools of New York City. 

As you read each test question and its underlined answer, judge 
the extent to which the knowledge or academic skill tested in the 
question is relevant to competent performance as a beginning 
teacher of Early Childhood Education New York City schools . 
There are four response categories: 

1. Crucial 

2. Important 

3 • Questionable 
4. Not Relevant 

When you have made your judgment, locate the column on the Job 
Relevance Form with the appropriate heading and fill in the 
corresponding oval (C, I, Q, N) with a heavy, dark mark so that 
you cannot see the letter in the oval. (Please use the pencil 
given to you by the project staff.) Before you mark a space, 
please make sure that the number on the form matches the number 
of the question in the question booklet. If you wish to change a 
response, erase your first choice completely so that your final 
judgment will be the only one picked up by the scanning machine. 

In making your judgments you are not to be concerned about how 
many questions you are assigning to each category; your 
responsibility is to apply your best judgment in evaluating each 
question individually. 

After you have finished making your judgments about the questions 
on a page, and again when you have finished the entire question 
booklet, please look over the questions and your responses to be 
sure that you are satisfied with your judgments. Also, check 
that the number of the last question for which you have recorded 
a judgment on the form corresponds to the number of the last 
question in the question booklet . Because the same form is being 
used for all tests, there may be more questions listed on the 
form than there are in the question booklet with which you are 
working. 



Used with copyright permission of the Educational Testing Service. 
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NTE Specialty Area Test in Early Childhood Education 
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Appendix K 



TEST COMPREHENSIVENESS CC*u*2NT SHEET 
NTE Specialty Area Test In Early Childhood Education 



Use the space below to list any major content topics in your 
institut n's teacher preparation sequence that do not appear to 
be covert i in the Test Content Description. Please note that the 
topics you write in s\ould be of approximately the level of 
specificity as those listed. Do not list topics that may be 
subsumed under one of the categories listed- If you are uncertain 
whether to list a topic here, please wait until you have examined 
the packet of test questions. 



Date Panel Member Signature 

Used with copyright pet mission of the Educational Testing Service. 
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OVERVIEW OF TASKS TO BE PERFORMED BY MEMBERS OF 
THE KNO /LEDGE ESTIMATION PANEL 

The study in which you have been asked to participate is being 
conducted to determine the valiuity of the NTE Specialty Area 
Test in Early Childhood Education. One purpose of the study is 
to estimate the test performance of minimally knowledgeable 
candidates for certification as beginning teachers in the public 
schools of New York City. 

You have been selected to serve on the Knowledge Estimation 
Panel. Your task, as a member of this panel, will be to make 
judgments about the difficulty of individual test questions for 
persons who have the minimum levels of knowledge and academic 
skills necessary for competent per forme ace as a beginning teacher 
in New York City ^ Your judgments will be combined with judgments 
made by other panel members to derive an estimate of the probable 
test performance of this group of persons. 

School and college personnel from throughout the City are being 
assembled to make their judgments. The judgments, however, will 
be made individually and independently. Members of the same 
panel will not confer as a group, nor will they be informed of 
the judgments made by other members. The judgments of all 
members of a panel will be combined statistically to arrive at a 
summary judgment for the ^<*nel about each test question. The 
summary results for the test questions also will be comb i^d, and 
the final summary results will be published in a report 
describing the study and its findings. 

Several items are intended to help you prepare for your tasks: 



1. The Test Content Description identifies the major 

groups of topics covered by the test and indicates the 
relative emphasis given to each topic. It is provided 
to familiarize you with the general content of the test 
before you see the test questions themselves. 



2. The Knowledge Estimation Form is to be used to record 
your judgments about the test questions. Please study 
the instructions for completing the form. 



(continued) 

Used with copyright permission of the Educational Testing Service. 
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3. The exercise entitled "Examinee Knowledge About 
Illustrative NTE Test Questions" \ ill be used at the 
panel meeting. This exercise will provide you with 
some experience in making judgments about the knowledge 
that characterizes defined groups of examinees for 
questions that differ in difficulty. 

4. A Panelist's Comment Sheet for your use is also 
included . 

You have been asked to participate in this study because you are 
familiar with the knowledge and academic skills needed for 
competent perf ormar^e as a beginning teacher of Early Childhood 
Education in New \ork City. Your qualifications to participate 
in this study are an important part of the study methodology. In 
order that the final report is as informative as possible, we are 
asking for your permission to identify you in the report. A 
consent form is enclosed and will be collected. In the final 
report, your name will not be associated with your individual 
judgments. 
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INSTRUCTIONS FOR COMPLETING THE KNOWLEDGE ESTIMATION FORM 

Please print your full name and darken the corresponding ovals. 
The other information is NOT necessary. 

Your task is to make judgments abo it che difficulty of individual 
test questions for minimally knowledgeable teachers of Early 
Childhood Education. You will be risked to draw upon your own 
experience to construct a hypothetical group of persons, each of 
whom, in your judgment, has the ninimum levels of knowledge and 
academic skills necessary for competent performance as a 
beginning teacher of Early Childhood Education in New York City, 

In drawing upon your experience, you will probably identify at 
least three types of persons for whom different levels of 
achievement would represent minimum levels of knowledge and 
academic skill: (1) those who will pursue non-school related 
careers after graduation; (2) those who will pursue careers in 
elementary schools; and (3) those who will pursue graduate study 
full-time. This study is concerned only with persons who will 
pursue careers in Early Childhood Education. Your assessments 
with respect to minimally knowledgeable persons should be made 
with only group "(2)" in mind . 

As you read each test question and its underlined answer, think 
of this group. Judge what percentage of the students \n the 
group would be able to identify or arrive at the answer to the 
question. If there were 100 minimally knowledgeable students, 
how many of them would know the answer? In making your estimate, 
assume that the students would not guess blindly if they did not 
know the answer. You should estimate the percentage who would 
know the answer without considering the possibility that some 
additional students might pick the answer purely as a lucky 
guess. 

Whe you have made your estimate, locate the column on the 
Knowledge Estimation form with the percentage heading that is 
closest to your estimate and lill in the corresponding space on 
the form with a heavy, dark mark so that you cannot see the 
letter in the oval. (Please use the pencil given to you by 
project staff.) Before you mark an oval, pl<**^e make sure that 
the number on the form matches <-he number oi _ne question in the 
question booklet. If you wish to change a response, erase yc z 
first choice completely so that your final judgment will be the 
only one pic iced up by the scanning machine. 

(continued) 

Used with copyright permission of the Educational Testing Service. 
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If you feel that your experience provides you with no basis for 
making an estimate about a question, you may fill in the 
corresponding oval labeled DNK (for "Do Not Know"). The DNK 
category is not to ha used simply because you have difficulty in 
deciding upon a percentage estimate; you are to make a decision 
even if it is a difficult one. The DNK category is to be used 
only when you have no basis for making an estimate. 

I'l making your judgments you are not to be concerned about how 
many questions you are assigning to each category; your 
responsibility is to apply your best judgment in evaluating each 
question individually. 

After you have finished making your judgments about the questions 
on a page, and again when you hav^ finished the entire question 
booklet, please look over the questions and your responses to be 
sure that you are satisfied with your judgments. Also, check 
tha* the number of the last question for which you have recorded 
a judgment on the form corresponds to the number of the last 
question in the question booklet . Because the same form is being 
used for all tests, there may be more questions listed on the 
form than there are in the question booklet with which you are 
working. 
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Appendix O 



EK 
TEKE 

EXAMINEE KNOWLEDGE ABOUT ILLUSTRATIVE NTE TEST QUESTIONS 
An EXERCISE for Members of the Knowledge Estimation Panel 



As a member of a Knowledge Estimation Panel, you will be asked to 
draw upon your experience to construct a hypothetical group of 
persons, each of whom, in your judgment has the minimum levels of 
knowledge and academic skills for competent performance as a begin- 
ning teacher in New York City. To help you prepare for your task, 
you will be asked to participate in an exercise in which you will 
estimate the performance of a national sample of NTE examinees on 
a series of test questions. The questions (to which you will be 
given the answers) are drawn from the Core Battery tests, which 
are taken by most NTE examinees regardless of the fields in which 
they plan to teach. For each test question, you will be asked to 
estimate the percentage of examinees in two categories who knew 
the answer. 

The first category of examinees include those whose scores, while 
not the lowest, were below the average for the total group (between 
the 20th and 40th percentiles). The second category contains 
examinees whose scores were above average for the total group 
(between the 60th and 80th percentiles). After you have made your 
estimates for \:he two groups, you will be given the "actual" per- 
centages of those ho selected the correct answers. 

You will also be asked to estimate the performance of the hypotheti- 
cal group of persons you have constructed for this task — those who 
have the minimum levels of knowledge and academic skills for com- 
petent performance as a beginning teacher in New York City. 

You may record your estimates on the form provided for this purpose. 
When you have made your estimate, locate the column on the form with 
the percentage heading that is closest to your estimate and fill 
in the corresponding oval on the form. Since the questions repre- 
sent a variety of subject matter fields, your teaching experience 
may give you little or no basis for estimating student knowledge 
about some of the test questions. If you feel you have no basis 
for making an estimate about a question, fill in the corresponding 
oval labeled DNK (for "Do Not Know"). 

The exercise is not intended to help you formulate your conception 
of the minimally knowledgeable student; rather, it is designed to 
give you some experience in making consistent judgments about the 
knowledge demonstrated by defined groups of examinees for questions 
that differ in difficulty. 



Used with copyright permission of the Educational Testing Service. 
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THE NEW REGULATIONS REQUIRE THAT IN RAINY WEATHER 
STUDENT CROSSING-GUARDS WEAR RAINCOATS, HATS, AND BOOTS 
PONCHOS MAY BE SUBSTITUTED FOR RAINCOATS AND HATS. IN THE 
EVENT OF SNOW, THESE ST IvENTS MUST WEAR GLOVES IN 
ADDITION TO 1HE OUTERWEAR REQUIRED ABOVE. 

ON A SNOWY DAY, WHICH OF TF FOLLOWING COMBINATION 
OF OUTERWEAR WOULD MEET THi. NEW REGULATIONS? 

I. RAINCOATS, HATS, BOOTS, AND GLOVES 

II. PONCHOS, GLOVES, AND BOOTS 

III. PONCHOS, HATS, AND GLOVES 



(A) I ONLY 

(B) II ONLY 

(C) III ONLY 

(D) I AND II ONLY 

(E) I, II, AND HI 
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2 . MANY STUDENTS WHO WERE OUTSTANDING IN THEIR HIGH SCHOOL 
STUDIES FIND IT MORE DIFFICULT TO ADJUST TO COLLEGE THAN 
DO THOSE WHO WERE ACADEMICALLY AVERAGE OR THOSE WHO 
DID POORLY IN HIGH SCHOOL; THE REASON IS THAT MANY OF 
THESE FORMERLY OUTSTANDING STUDENTS FIND IN COLLEGE NOT 
ONLY THAT THERE ARE ANY NUMBER OF STUDENTS WHO ARE AS 
OUTSTANDING ACADEMICALLY AS THEY ARE, BUT ALSO THAT THERE 
ARE A LARGE NUMBER WHO ARE EVEN MORE GIFTED. 

WHICH OF THE FOLLOWING BEST SUMMARIZES THE MAIN IDEA OF 
THIS STATEMENT? 

(A) SOME STUDENTS WHO WORKED HARD AND WHO WERE 
ACADEMICALLY OUTSTANDING IN HIGH SCHOOL DO NOT WORK 
AS HARD IN COLLEGE 

(B) SOME STUDENTS WHO SHOWED THEMSELVES TO BE OF 
AVERAGE OR LOW ABILITY IN THEIR HIGH SCHOOL WORK ARE 
HAPPIER IN COLLEGE THAN THEY WERE IN HIGH SCHOOL. 

(C) MANY STUDENTS WHO DID WELL IN HIGH SCHOOL DO NOT 
SEEM TO L.EARN AS MUCH ONCE THEY GET TO COLLEGE 

(D) MANY STUDENTS DO NOT DO AS WELL IN COLLEGE AS THEY 
DID IN HIGH SCHOOL BECAUSE THEY ARE UPSET BY THE 
LARGE NUMBERS OF STUDENTS IN THEIR CLASSES. 

(E) MANY OUTSTANDING STUDENTS FIND THAT, BECAUSE OF THE 
HIGH LEVEL OF ACADEMIC COMPETITION, IT IS MORE 
DIFFICULT TO DO WELL IN COLLEGE THAN IT WAS IN HIGH 
SCHOOL. 
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• SENTENCE 1: I RECENTLY TOOK MY DAUGHTER TO THE VIRGIN 
ISLANDS ON VACATION. 

SENTENCE 2: THAT PART OF THE WORLD IS SURELY ONE OF THE 
MOST BEAUTIFUL. 

SENTENCE 3: THE DAY WE ARRIVED WAS SUNNY, THE 
TEMPERATURE WAS 82 DEGREES. 

SENTENCE 4: I DONT THINK THAT MY DAUGHTER, WHO IS ONLY 
S YEARS OLD, HAS EVER BEEN MORE EXCITED. 

SENTENCE 5: SHE SAID THAT SHE WANTED TO GO STRAIGHT TO 
THE OCEAN FOR A SWIM. 

IF THE WRITER IS NOT INTENDING TO DECEIVE HIS OR HER 
AUDIENCE, WHICH OF THE FOLLOWING DESCRIPTIONS OF THESE 
SENTENCES IS ACCURATE? 

(A) ONLY SENTENCES 1 AND 3 STATE FACTS. 

(B) SENTENCES 1, 3, 4, AND 5 STATE FACTS ONLY. 

(C) SENTENCES 2 AND 4 STATE OPINIONS AND NOT FACTS. 

(D) SENTENCE 4 STATES BOTH FACT AND OPINION. 

(E) SENTENCE 5 STATES BOTH FACT AND OPINION. 
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FOR A WRITER, THE RAREST PRIVILEGE IS NOT MERELY 

A B 

TO DESCRIBE HER COUNTRY AND TIME BUT TO HELP 
C 



SHAPE II. NO ERROR 
P. E 
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TliE AGENT. PA SSING THROUGH THE CROWD WITHOUT BEING 
ISIQTIfTP BY HARDLY ANYONE. 

(A) THE AGENT, PASSING THROUGH THE CROWD WITHOUT BEING 
NOTICED BY HARDLY ANYONE. 

(B) THE AGENT PASSED THROUGH THE CROWD WITHOUT HARDLY 
BEING NOTICED BY ANYONE. 

(C) THE AGENTS PASSIM; IKOICII Tilt CROWD WAS NOT HARDLY 
NOTICED BY ANYONE. 

(D) NO ONE HARDLY NOTICED HOW THE AGENT PASSED THROUGH 
THE CROWD. 

(E) THE AGENT WAS HARDLY NOTICED AS SHE PASSED THROUGH 
THE CROWD. 
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Estimates of Percentages of Examinees in Two Score Categories 
Who Knew the Answer to Illustrative Questions 



Below Average Actual 
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